System and method for processing subpicture data

ABSTRACT

A method is described for rendering subpicture images comprising: reading a subpicture image from a digital versatile disk (“DVD”); identifying regions within the subpicture image for which α is below a specified threshold value; and rendering on a video screen only those regions of the subpicture image for which α is above the specified threshold value.

BACKGROUND

1. Field of the Invention

This invention relates generally to the field of multimedia playbacksystems. More particularly, the invention relates to a system and methodfor efficiently decoding, converting and rendering DVD subpicture dataon a multimedia playback system.

2. Description of the Related Art

The MPEG-2 standard is one of the most popular audio/video compressionstandards and is used in a wide range of multimedia applications. Forexample, MPEG-2 is used as the underlying compression algorithm for bothdigital cable and satellite broadcast channels and for Digital VersatileDisks (“DVDs”) (also referred to as “Digital Video Disks”). In addition,MPEG-2 is used as the compression algorithm for a vast number ofcomputer-based multimedia applications.

Although the underlying MPEG-2 compression algorithm is similar for eachapplication, the manner in which the compressed audio/video content ispacketized, transmitted, stored and otherwise processed variesconsiderably from one application to the next. For example, cable andsatellite companies transmit the MPEG-encoded channels within amultiplexed “transport streams” comprising transport packets of a fixed188 bytes in length. Transport streams follow the T-STD (transportsystem target decoder) model defined in ISO/IEC 13818-1.

By contrast, MPEG content is stored on DVDs in a unique “program stream”format, using packets of 2048 bytes in length. Normal program streamsfollow P-STD (program stream target decoder) model defined in ISO/IEC13818-1. The DVD specification defines a new target decoder model, E-STD(extended system target decoder), which is an extension of the P-STDmodel and which is outside of the MPEG-2 specification. Morespecifically, the DVD specification defines additional media types anddata structures that are stored and transmitted within MPEG data packetsbut are not part of the MPEG specification including, for example,sub-picture data defining graphics to be displayed on thetelevision/computer screen (e.g., interactive navigation menus,subtitles, overlays, . . . etc), AC-3 audio, Presentation ControlInformation (“PCI”) used to navigate a play path through DVD multimediafiles, and Data Search Information (“DSI”) used to provide differenttypes of playback timing information. In addition, DVD defines a largeamount of sideband metadata that is not contained within the MPEG streamitself but which is used to locate content within the DVD given a set ofuser selections and/or predefined navigation parameters.

As such, DVD players must be equipped with the hardware and/or softwarecapable of decoding these additional media types and metadata in orderto create a compliant DVD viewing experience. By contrast, the hardwareand/or software used in typical cable and satellite set-top boxes(“STBs”) are only capable of decoding standard MPEG-2 transport streams.

Given the increasing popularity of DVDs and the fact that most usersalready subscribe to cable and/or satellite services, it would bepractical to develop a multimedia system which would allow DVDs and(potentially) other media types to be properly decoded and renderedusing a standard transport stream decoder.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained from thefollowing detailed description in conjunction with the followingdrawings, in which:

FIG. 1 illustrates a network architecture for implementing embodimentsof the invention.

FIG. 2 a illustrates one embodiment of a home media server hardwarearchitecture.

FIG. 2 b illustrates one embodiment of a home media server softwarearchitecture.

FIG. 3 illustrates a plurality of media modules installed in oneembodiment of a home media server.

FIG. 4 illustrates one embodiment of a home media server which includesa DVD/CD/CD-RW drive.

FIG. 5 illustrates a home media server communicating to two audio/videomedia nodes over a wireless network.

FIG. 6 a illustrates one embodiment of a multimedia node hardwarearchitecture.

FIG. 6 b illustrates one embodiment of a multimedia node softwarearchitecture.

FIG. 6 c illustrates one embodiment of the system for tuning betweenmultimedia channels.

FIG. 6 d illustrates one embodiment of the system for playing a CDjukebox.

FIG. 7 a illustrates one embodiment of a home media server forcoordinating between standard telephone services and IP telephoneservices.

FIG. 7 b illustrates one embodiment of a software architecture forimplementing standard telephone, IP telephone and/or video conferencing.

FIG. 7 c illustrates a signal diagram of communication between a homemedia server and telephone.

FIG. 8 a illustrates a home media server coordinating between standardbroadcast channels and packet-switched channels (e.g., the Internet).

FIG. 8 b illustrates a potential progression of bandwidth allocationbetween packet switched channels and analog/digital broadcast channels.

FIG. 8 c illustrates multimedia buffering according to one embodiment ofthe invention.

FIG. 9 a illustrates a histogram showing a normalized bitrate for aparticular multimedia stream.

FIG. 9 b illustrates one embodiment of a system for intelligentbandwidth allocation and buffering.

FIG. 9 c illustrates bitrate data normalized at one second intervals.

FIG. 9 d illustrates bitrate data for three separate multimedia streamsnormalized at 10 second intervals.

FIGS. 9 e–m illustrate histograms of normalized bitrate data for variousDVDs.

FIG. 10 illustrates a cable television module according to oneembodiment of the invention.

FIG. 11 illustrates a satellite module according to one embodiment ofthe invention.

FIG. 12 illustrates a cable modem module according to one embodiment ofthe invention.

FIG. 13 illustrates copyright tags implemented in one embodiment of theinvention.

FIG. 14 illustrates transport stream and program stream generation frompacketized elementary streams.

FIG. 15 illustrates the manner in which audio, video and subpictures arestored on a digital versatile disk (DVD).

FIG. 16 a illustrates one embodiment of the invention for multimediastream conversion.

FIG. 16 b illustrates a content conversion module according to oneembodiment of the invention.

FIG. 16 c illustrates timestamp conversion and repacketization logicaccording to one embodiment of the invention.

FIG. 16 d illustrates a method for converting program stream packets totransport stream packets according to one embodiment of the invention.

FIG. 17 illustrates seamless angles on a DVD where two video streamsshare a common time base.

FIG. 18 illustrates non-seamless angles on a DVD where two video streamsuse unrelated time bases.

FIG. 19 illustrates fast-forward and rewind techniques employed by oneembodiment of a stream processing system.

FIG. 20 illustrates a mechanism for performing fast forward and rewindof DVD content on a transport processor.

FIGS. 21 a–b illustrate issues surrounding displaying still video imagesusing a transport processor.

FIG. 22 illustrates one embodiment of the invention for processing AC-3audio content.

FIG. 23 illustrates subpicture processing according to one embodiment ofthe invention.

FIG. 24 illustrates a method implemented by a subpicture processoraccording to one embodiment of the invention.

FIG. 25 illustrates subpicture processing according to one embodiment ofthe invention.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however, toone skilled in the art that the invention may be practiced without someof these specific details. In other instances, well-known structures anddevices are shown in block diagram form to avoid obscuring theunderlying principles of the invention.

Introduction

An advanced digital recorder system (hereinafter “Media Server”) isdescribed in the co-pending application entitled MULTIMEDIA ANDCOMPUTING SYSTEM, Ser. No. 09/653,964, Filed Sep. 1, 2000, which isassigned to the assignee of the present application and which isincorporated herein by reference. Certain aspects of this advanceddigital recorder system will now be described followed by a detaileddescription of embodiments of a system and method for processingsubpicture data. It should be noted, however, that the underlyingprinciples of the invention may be implemented on virtually any type ofdigital multimedia receiver system. For example, the embodiments of theinvention described below may be employed on both advanced multimediarecorder systems such as a “Media Server” and standard cable andsatellite set-top boxes (e.g., such as those described above in thebackground section).

Embodiments of a Media Server

As illustrated in FIG. 1, in one embodiment of the invention, a digitalmedia server 110 equipped with a processor and a mass storage deviceacts as a central repository for decoding, storing and distributingmultimedia content and data. More particularly, the digital media server100 coordinates multimedia content from Internet communication channels120 (e.g., DSL, cable Internet), broadcast communication channels 130(e.g., digital/analog cable, satellite), and/or Public SwitchedTelephone Network (“PSTN”) communication channels 170 (i.e., standardtelephone) to provide a stable, real-time home media network 140 for aplurality of network devices 150–151, 160–166.

As illustrated in FIG. 2 a, one embodiment of a home media server 110computing architecture includes a central processing unit 200 capable ofprocessing data and multimedia content stored in main memory 201 and amass storage device 230 for storing data and multimedia content. In oneembodiment, the central processing unit 200 is a Pentium®-classprocessor such as a Pentium III® operating at a 1 GHz or faster clockfrequency. It should be noted, however, that the underlying principlesof the invention are not limited to any particular processor speed orprocessor type. The main memory 201 may be a random access memory or anyother dynamic storage medium (e.g., SDRAM, DDRAM, RD-RAM, . . . etc).The mass storage device 230 of one embodiment is capable of storinghundreds, or even thousands of hours of multimedia content (e.g.,movies, digital audio, . . . etc) as well as other types of digital data(e.g., computer programs, word processing documents, . . . etc). Devicestransmit and receive data to/from the mass storage device 230 over ahigh speed interface such as an enhanced IDE interface with Ultra DMAcapabilities or a Small Computer System Interface (“SCSI”). However,various other interfaces may be employed while still complying with theunderlying principles of the invention.

An application-specific integrated circuit (“ASIC”) 210 coordinatescommunication between the various system components and offloads certaindesignated processing tasks from the CPU. The ASIC may be custom builtbased on the requirements of the home media server 110 or may be builtusing gate arrays, standard cells or programmable logic devices.

Communication modules 240–245 electrically coupled to the home mediaserver 110 via a system bus 220, allow the home media server 110 tocommunicate over different local and remote communication channels. Inone embodiment, the system bus 220 is a peripheral componentinterconnect (“PCI”) bus, although various other bus types may beconfigured within the home media server 110 (e.g., ISA, EISA, MicroChannel, VL-bus . . . etc).

In the particular embodiment illustrated in FIG. 2 a, the communicationmodules 240–245 electrically coupled to the system bus 220 include an RFnetwork module 240 for communicating over the home media network 140(i.e., via a wireless RF channel), a cable TV module 241 for receivingbroadcast cable channels, a cable modem module 242 for providingInternet access via a cable system (i.e., using the TCP/IP protocol), asatellite TV module 243 for receiving satellite broadcasts, and a DSLmodule 244 for DSL Internet access. Moreover, a virtually unlimitednumber of new modules may be added as necessary to support new orexisting communication channels/protocols (as indicated by module 245).

Other components within the home media server 110 architecture includean MPEG-2 decode module 202 (and/or other decode modules such as AC3,MPEG-1, . . . etc); an audio module 203 comprised of a digital-to-analogconverter, a Sony-Philips Digital Interconnect Format (“SP-DIF”)interface and a standard telephony interface for providing digital andanalog audio and standard telephone service to external audio/telephonydevices; an Ethernet port provided directly the system ASIC 210 (asindicated by the “100 Base-T Ethernet” designation); a Firewire (IEEE1394) port 204; a Universal Serial Bus (“USB”) port 205; and an infraredport 206. Various other communication interfaces may be configured inthe system, either directly on the primary home media serverarchitecture 110 (e.g., on the media server 110 “motherboard”), or as anadd-on module 240–245. Moreover, the communication modules (e.g.,202–206), the CPU 200 and/or the memory 201 may be incorporated withinthe system ASIC 210, rather than as separate modules as illustrated inFIG. 2 a.

Embodiments of the home media server 110 may also be equipped with a DVDdrive, CD player, CD Read-Write drive, recordable DVD drive (asdescribed in greater detail below), and/or any other type of portablestorage medium 235. In one embodiment, these devices may communicatewith the home media server 110 via an AT Attachment Packet Interface(“ATAPI”), although the type of interface used is not pertinent to theunderlying principles of the invention.

FIG. 2 b illustrates a software architecture employed in one embodimentof the home media server 110. Different hardware architectures 280 maybe used to support the software, including the hardware architectureillustrated in FIG. 2 a. A multitasking, multithreaded operating system(“OS”) 270 (e.g., Linux, UNIX, Windows NT®) with real time streamingsupport is executed on the hardware 280. In one embodiment, certainproprietary customizations 292 are programmed in the media server OS 270including, for example, real time services for streaming audio and video(real time OSs typically do not include these features).

A media toolkit 260 executed within the home media server 110 providesan application programming interface (“API”) for the different mediaserver 110 applications described herein as well as a set of mediaserver 110 utilities. In one embodiment, a minimum quality of service(“QoS”) is defined within the media toolkit 260. The media toolkit 110may be comprised of several functional layers including a media streamabstraction layer; a media stream resource management layer; asecurity/conditional access layer; and a transport/storage abstractionlayer.

In one embodiment, the home media server 110 is configured with supportfor the realtime transport protocol (“RTP”) and the realtime streamingprotocol (“RTSP”) (see, e.g., RTP/RTSP module 251). Briefly, RTP is anIP protocol which supports real time transmission of voice and video. AnRTP packet typically resides on top of a user datagram protocol (“UDP”)and includes timestamping and synchronization information in its headerfor proper reassembly at the receiving end. RTSP is a well knownprotocol for streaming multimedia content over a network. It should benoted, however, that various other streaming protocols may beimplemented while still complying with the underlying principles of theinvention (e.g., Quicktime®, Windows Media, . . . etc).

RTP and RTSP were designed primarily for PC to PC communication overnon-realtime networks. Accordingly, because one embodiment of the homemedia server 110 operates over a realtime home media network 140 andcommunicates to devices other than PCs (e.g., media nodes 150, 151),certain optimizations 290 may be programmed within the RTP/RTSPcomponent 251. These include, for example, support for multiple datastreams between the home media server 110 and the various multimediadevices 150, 151; 160–166 (e.g., one or more video streams and severalaudio streams)

Apache HTTP server support 250 is also included in one embodiment of thehome media server 110. Apache is an open-source HTTP server softwareproduct which may be executed on various modern desktop and serveroperating systems. Once again, certain media server customizations 290may be included within the Apache component 250, further refining HTTPsupport for the home media network 140 environment.

In one embodiment, a plurality of device servers 252 are executed on thehome media server 110. These devices servers provide application supportfor each of the media nodes 150–151 and other devices 160–166communicating with the home media server 110 over the home media network140. For example, in response to a “tune” command sent from a media node150, a tuning device server will cause the home media server to tune toa broadcast channel specified by the tune command. The tuning server mayalso include conditional access functionality (i.e., only allowingtuning to occur for channels to which the user has access rights).

Although described above as a “software” architecture, it should benoted that various elements illustrated in FIG. 2 b may also beimplemented in firmware and/or hardware while still complying theunderlying principles of the invention.

Referring again to FIG. 1, numerous digital and analog devices may beconfigured to communicate with the home media server 110 over the homemedia network 140. By way of example, and not limitation, these includepersonal computers 160, cameras or digital camcorders 161, printers 162,notebook computers 163, automotive audio/video systems 164, cell phonesor personal digital assistants 165, standard telephones 165 (includingfax machines), home security systems (not shown); and/or home climatecontrol systems (not shown).

In one embodiment, complex multimedia and data processing such as tuningto selected channels, recording of specified programs/music, storingphone numbers and personal data, connecting to remote network sites,etc., is performed at the media server 110, rather than at theindividual network devices 160–166. As such, these devices 160–166 maybe manufactured relatively inexpensively. For example, multimedia nodes150, 151 may be equipped with just enough processing power and memory toreceive and play back a multimedia signal, with storage and control(e.g., tuning) functions offloaded to the home media server 110.Similarly, a telephone 166 may be designed with nothing more than a lowpower microcontroller coupled to an RF transmitter, with telephonyfunctions and contact data storage provided by the home media server110. In addition, because these network devices 160–166 do not requireas much circuitry, they will be lighter than currently-existing devices,and will consume less power.

In one embodiment, the primary communication medium over which the homemedia server 110 and the various devices 150–151, 160–166 communicate iswireless RF (e.g., via network module 240), with terrestrial transportconnections such as Ethernet reserved for devices which are not withinRF transmission range. Moreover, certain devices which require asubstantial amount of home media network 140 bandwidth (e.g., highdefinition television 170), and/or devices which are in close proximityto the media server 110 may be configured to communicate overterrestrial transports, depending on the requirements of the particularconfiguration.

One Embodiment of a Multimedia Node Architecture

Distributed multimedia nodes 150 and 151 illustrated in FIG. 1 providean interface to the home media network 140 for audio systems 170 (e.g.,audio amplifiers and speakers) and/or video systems 171 (e.g., standardtelevision sets, wide screen television sets, high definition television(“HDTV”) sets, or any other device capable of displaying video).

As illustrated in FIG. 6, one particular embodiment of a multimedia nodearchitecture (see, e.g., node 150 in FIG. 1), is comprised of a networkinterface 605 coupled to the multimedia node bus 610 (e.g., a PCI bus);a system ASIC 620, including MPEG-2 decode/graphics logic 630 (or othermultimedia decode logic); a central processing unit 640 and memory 650;an audio processing unit 660; and/or an infrared port 670.

The local central processing unit 640 and memory 650 execute programsand process data locally (i.e., at the multimedia nodes 150, 151). Thenetwork interface 605, which may be an RF interface or a terrestrialinterface (e.g., Ethernet), receives/transmits multimedia content andcontrol data from/to the home media server 110 over the home medianetwork 140. The system ASIC 620 decodes and processes incomingmultimedia content transmitted from the home media server 110 via MPEG-2decode/graphics logic 630 (or other multimedia compression standard) toproduce one or more video outputs 680 (e.g., an NTSC output, an HDTVoutput, . . . etc). In one embodiment, a separate audio processing unit660 produces both digital and analog audio outputs, 681 and 682,respectively. Moreover, one embodiment also includes a local massstorage device (not shown) for storing certain multimedia content and/ordata (e.g., frequently-requested content/data).

In one embodiment, the local infrared interface 670 receives controlcommands from a remote control unit (e.g., unit 532 in FIG. 5) operatedby a user. As described below with reference to FIG. 5, controldata/commands received through the infrared interface 670 maysubsequently be transmitted to the home media server 110 for processing.For example, in one embodiment, if a user selects a command to change toa live broadcast channel (e.g., the evening news), the command istransmitted from multimedia node 151 to a home media server 110 tuningapplication, which forwards the command to a video module (e.g., CATVmodule 241, satellite TV module 243). The tuner in the video module(see, e.g., tuner 910 in FIG. 9) then tunes to the requested frequencyand the new video signal is streamed from the home media server 110 tothe multimedia node 151. Alternatively, or in addition, if the requestedcontent was previously stored on the home media server 110 (e.g., onmass storage device 230), then no broadcast tuning is necessary—thecontent is simply read from the storage device 230 and transmitted tothe multimedia node 150. Accordingly, as described in greater detailbelow, the amount of audio/video content which must be broadcast to thehome media server 110 (i.e., over dedicated broadcast channels) issignificantly reduced.

One embodiment of a multimedia node 150 software architecture isillustrated in FIG. 6 b. Various hardware architectures 690 may be usedto support the software architecture, including the architectureillustrated in FIG. 6 a. An operating system 691 executed on hardware690 (e.g., Windows '98, Linux, . . . etc) includes client customizationsfor optimizing communication over the home media network 140 (e.g.,providing low level support for real time streaming of audio and video).

A standard set of user interface components 694 included in oneembodiment may be employed (e.g., by application developers) to generateunique interactive interfaces at each of the media nodes 150, 151. Forexample, a user-navigable tuning index may be included which listsavailable content by dates/times and allows users to graphically selecta particular broadcast channel and/or stored content from the massstorage device 230.

In addition, support for the hypertext markup language (“HTML”) and/orthe extensible markup language (“XML”) are included in one multimedianode 150 embodiment, allowing users to download, view and modify varioustypes of Internet content (e.g., Web pages).

Applications executed on multimedia nodes 150, 151 may includeclient-specific applications 697 and/or home media server 110 controlapplications 696. Two examples of media server control application 696will now be described with reference to FIGS. 6 c–6 d. It should benoted, however, that these examples are for the purpose of illustrationonly and should not be read to limit the scope of the invention.

FIG. 6 c illustrates an embodiment for tuning to a particular broadcastchannel (e.g., cable, satellite broadcast). A user initially selects aparticular channel (e.g., via a remote control device) to be viewed atthe local multimedia node 151. The command is received by the local TVtuning application 601 executed on the multimedia node 151, which sendsa tuning request to the home media server 110 over the home medianetwork 140. A tuner server module 602 executed on the home media server110 receives the request and forwards the request to the tuner driver271 (e.g., through the media toolkit 260). In response to the command,the tuner driver 271 directs the tuner hardware (see, e.g., tuner 910 inFIG. 9 described below) to tune to the specified channel. The tunerdriver also communicates with the conditional access subsystem (see,e.g., module 930 in FIG. 9) to determine whether the user has the rightto view the particular channel (e.g., the channel may be asubscription-based channel such as HBO).

If the user has access rights, then the tuner server module 252 createsvideo stream for the specified broadcast channel and feeds the stream tothe multimedia node 151. In one embodiment, the tuner server module 602creates the stream using the RTP/RTSP protocol (i.e., by communicatingwith the RTP/RTSP module 251). The media node receives the streamthrough the local TV tuning application 601 and decodes the stream usinga codec embedded within the media toolkit 260 (various additionalencoding features of the system are set forth below). Once decoded, thevideo image is generated on the video display.

FIG. 6 d illustrates an embodiment for employing a music jukebox at amultimedia node 150. A user initially selects a particular music playlist (e.g., via a remote control device) to be played at the multimedianode 150. The request is received by a music jukebox application 601executed on the multimedia node 150, which forwards the request to thehome media server 110. A jukebox server module 604 executed on the homemedia server 110 receives the request and forwards the request to themedia server file system 606 (through the media toolkit 260), whichretrieves the music files from the mass storage device 230. The jukeboxserver module 604 then creates the audio stream for the specified filesand feeds the stream to the multimedia node 150. In one embodiment, thetuner server module 252 uses the RTP/RTSP protocol to create the stream(e.g., through communication with the RTP/RTSP module 251). Themultimedia node 150 receives the stream through its jukebox application601 and decodes the stream using a codec embedded within the mediatoolkit 260. Once decoded, the music tracks are reproduced on a set ofspeakers.

Network Operations Center

In one embodiment, a network operation center (“NOC”) 180 illustrated inFIG. 1 provides data and multimedia services for the home media server110. The NOC may be comprised of one or more servers communicating withthe home media server 110 over the Internet 120 (or other network). Inone embodiment, the NOC performs one or more of the following functions:

Remote Monitoring and Maintenance. The NOC 180 monitors the home mediaserver to ensure that it is operating within normal parameters. In oneembodiment, the home media server 110 periodically transmits a statusupdate to the NOC 180, indicating whether any particular services arerequired and/or whether any problems have arisen. If the home mediaserver 110 fails to transmit a status update after a predeterminedperiod of time, the NOC 180 may take steps to determine the cause of theproblem and/or take steps to contact the user. For example, diagnostictests may be performed to evaluate the network connection between thehome media server 110 and the NOC 180. If the diagnostic tests do notprovide an answer, then the user may be contacted via telephone todetermine the next course of action. In one embodiment the NOC 180 isstaffed by engineers or other technical assistance personnel 24-hours aday.

In one embodiment, the user's home security system and/or fire systemcommunicates with the home media server 110 over the home media network140. Accordingly, the home media server 110 may be programmed to relayhome security and/or fire system data to the NOC 180 on a periodic basisduring certain periods of time (e.g., every minute). If a securityproblem or fire trouble indication is encountered during this period oftime (e.g., a door/window is opened without the proper security accesscode), the NOC 180 may notify a local law enforcement agency and/or thelocal fire department.

Automatic Downloading and Upgrades. In one embodiment, the NOC 180provides automated software downloading and upgrades to the home mediaserver 110. For example, when a new software interface/program (e.g., anew graphical user interface) is developed for the home media server 110it may be automatically downloaded from the NOC 180 and installed.Whether the NOC 180 should upgrade software on the home media server 110automatically may be determined by each individual user. Certain usersmay opt to manually upgrade all of their software while other users mayspecify a particular subset of software to be automatically upgraded.For example, a user may specify automatic upgrades for graphical userinterface software and manual upgrades for email clients and wordprocessors.

In addition, the NOC 180 may store up-to-date drivers for various homemedia network devices 160–166. In one embodiment, when a user purchasesa new device, such as a new telephone, the device may automaticallyidentify itself to the home media server 110 (e.g., via USB, IEEE 1394or other device identification technology). The home media server 110will then determine whether the device is supported by the system and,if so, query the NOC 180 for the latest driver. In one embodiment, oncethe driver is identified, it is automatically downloaded from the NOC180 and installed on the home media server 110. Once installed, thedriver may be updated automatically as new updates become available.Accordingly, driver software will no longer need to be bundled with homenetwork devices 160–166, resulting an additional cost savings to theend-user. In addition, users will not be required to install andconfigure device drivers for each new network device they purchase.

In addition, the home media server 110 may check the NOC 180 todetermine whether the home network device has been certified by the NOC180 as meeting some minimum level of quality and/or performance. In oneembodiment, device drivers will be downloaded and installed on thesystem only if the device has been certified. In this manner, a minimumquality of service may be maintained for all devices employed on thesystem.

The same type of device identification, certification, and driverinstallation may be performed for communication modules 240–245 such asthose described above. Thus, as with home network devices 160–166,certain communication modules 240–245 which do not meet a minimumquality of service requirement (e.g., those which perform inefficientbus transactions which tie up the system bus 220) will not be certifiedby the NOC 180 and will not be permitted to be installed in the homemedia server 110.

In addition, in one embodiment the communication modules 240–245 aremanufactured and certified by third party content providers (e.g.,satellite broadcast operators). In this embodiment, the contentproviders may configure the modules 240–245 to work with their ownproprietary communication protocols, encryption techniques, and/orapplication programs. For example, DirecTV™ may develop a proprietaryinterface for its users so that when a user selects his DirecTV feed hewill be presented with a unique user interface and/or channelprogramming environment. Accordingly, while the present inventionprovides a standard media transport interface for a variety of differentcommunication channels, service providers can still distinguish theirservices based on the proprietary applications/transports which theydevelop.

Logging/Data Warehousing. In one embodiment, the NOC 180 may performlogging and data warehousing for the home media server 110. Morespecifically, the NOC 180 may maintain a log of network transactions foreach home media server 110 and subsequently evaluate the log for avariety of reasons (e.g., to troubleshoot system problems, to determinea user's preferences and tailor services and/or advertising to thatuser, . . . etc). For example, by monitoring usage patterns, the NOC 180may determine that every time a certain Java applet is downloaded, thehome media server 110 crashes. As such, the NOC 180 may takes steps toensure that the applet in question is no longer downloaded by the homemedia server 110 (e.g., by notifying the user or automatically blockingthe applet). The NOC 180 could then notify the technical support staffto determine the problem with the applet.

Similarly, the usage log may be evaluated to determine the preferencesof a user and to provide specialized services to that user based onthose preferences. For example, based on the Web sites the user visitsand/or the channels that the user watches, the NOC 180 may determinethat the user is interested in baseball. As such, the NOC 180 mayautomatically provide baseball-related content to the user such as, forexample, broadcast schedules for upcoming games, a subscription offer toa sports magazine, advertisements, and various other baseball-relatedcontent. Similarly, the NOC 180 may determine that the user watchescertain television shows on a regular basis, and may automaticallydownload/record those shows on the home media server 110 (e.g., viaTCP/IP), so that they will be readily available for the user (e.g.,during non-broadcast periods of time).

Archiving. In one embodiment, users may backup multimedia content andother types of data at the NOC 180. For example, a user may take aseries of pictures with a digital camera and transmit the originals tothe NOC 180 for developing. In one embodiment, the NOC will transmit thepictures to a developer on behalf of the user and will store a backupcopy of each of the originals (the NOC 180 will be backed up regularlyto prevent loss of the originals).

In one embodiment, the NOC 180 will monitor all multimedia contentpurchased by the user over the Internet. For example, when the userdownloads a new compact disk (“CD”) from a music download site, the NOC180 will record the transaction in the user's profile. As such, the userdoes not need to store all of his/her multimedia content locally on thehome media server 110. Rather, because the NOC 180 keeps track of allthe content to which the user has access rights, the user can offloadstorage to the NOC 180 and re-download the content when necessary (e.g.,following a hard drive failure on the home media server 110).

Registration/configuration. In one embodiment, users may be required toregister with the NOC 180 upon purchasing a home media server 110 andrelated network devices 150–151, 160–166. The NOC 180 may prompt eachuser to respond to a series of question directed to the user'spreferences, the type/level of NOC 180 services desired by the user, theauthorization level of each user in the home media server 110 household(e.g., children may be provided with limited functionality), and anyother user-related data which may aid the NOC 180 in providing userservices. In one embodiment, the NOC 180 will automatically detect thehome media server 110 configuration and store this data in a userdatabase (e.g., to be used for software upgrades, troubleshooting, . . .etc).

Bridge to Satellite/Cable Operations. In one embodiment, the NOC 180will coordinate communication between the home media server 110 and anysatellite/cable services to which the user has subscribed. For example,the NOC 180 may forward pay-per-view requests transmitted from the homemedia server 110 to the various cable/satellite operators, and performthe back-end processing (e.g., authentication, billing) required for thepay-per-view transaction. Because a persistent communication channelexists between the NOC 180 and the home media server 110 (e.g., throughDSL or cable modem), no dial-up processing is required (unlike currentpay-per-view cable and satellite systems).

E-commerce Support. In one embodiment, rather than acting merely as aconduit between the home media server 110 and other e-commerce Internetsites, the NOC 180 may perform various e-commerce functions itself. Forexample, the NOC 180 may market and sell products on behalf of otheronline retailers (e.g., Amazon.com®). The NOC 180 of this embodiment mayperform the back-end processing (e.g., billing and record keeping)required for each e-commerce transaction.

Application Support. In one embodiment, the NOC 180 will provide supportfor applications executed on the home media server 110. For example, theNOC 180 may provide a compact disk database containing CD titles, trackinformation, CD serial numbers, etc. When a user copies his CDs onto themass storage device 230 of the home media server 110, the home mediaserver 110 may query the database (as described in greater detail below)and download title and track information (or other information) used toidentify/index each CD and each track. In one embodiment, public CDdatabases (e.g., such as the database maintained at “www.cddb.com”) maybe filtered and improved by the NOC 180 to remove errors. Various othertypes of application support may be implemented at the NOC 180consistent with the underlying principles of the invention.

Remote-Access Gateway. In one embodiment, the NOC 180 may be used as agateway to access to the home media server 110 from a remote location.For example, a user from a PC connected to the Internet may log in tohis home media server 110 through the NOC 180.

In one embodiment, the home media server 110 is configured to makeoutgoing connections only (i.e., to reject direct incoming connections).As described briefly above, the home media server 110 may poll the NOC180 periodically (e.g., every few seconds) and transmit a status update.During this periodic poll the home media server 110 may query the NOC180 to determine whether anyone is attempting to access the home mediaserver 110 from a remote location. If so, then the NOC 180 transmitsspecific information related to the connection attempt to the home mediaserver 110 (e.g., authentication data such as user ID's, passwords, . .. etc). The home media server 110 may then initiate a connection withthe remote user (e.g., using the remote user's IP address and encryptionkey).

In one particular embodiment, the NOC 180 will authenticate the remoteuser before notifying the home media server 110 of the connectionattempt. For example, upon receipt of a remote connection request, theNOC 180 may prompt the user to answer a series of questions (e.g.,personal questions, questions related to the user's account, . . . etc).If the user does not answer the questions in a satisfactory manner, amember of the NOC staff may contact the user directly (e.g., viastandard telephone or IP telephone). Regardless of how authenticationtakes place, once a remote user is authenticated, the user's data istransmitted to the home media server 110, which subsequently establishesa connection with the remote user.

Additional Home Media Server Embodiments and Applications

An external illustration of one embodiment of the home media server 110is shown in FIGS. 3 and 4 (from the back and front, respectfully). FIG.3 shows a home media server 110 with a plurality of different mediamodules 320–325 installed in its expansion bays (i.e., electricallycoupled to its system bus 220). As illustrated, in one embodiment,certain standard communication ports such as Ethernet 301, IEEE 1394302, USB 303, digital/analog audio 305, standard telephone 304, XGA/HDTV306, and/or other standard audio/video ports (e.g., AV output ports 307and 308) may be installed directly in the primary home media server,rather than as an expansion card.

As illustrated in FIG. 4, one embodiment of the system includes a DVDplayer 420 (and/or a CD player, CD-ReWritable drive, recordable DVDdrive or other type of portable digital media player/recorder). The DVDplayer 420 is capable of playing a DVD directly and/or transferringmultimedia content from the DVD to the mass storage device 230. In oneembodiment, as multimedia content is transferred to the storage device230, either from the DVD player 420 or over the Internet 120, anindexing module executed on the home media server 110 indexes themultimedia content in a content database (not shown).

Various DVD/CD identification techniques may be used to identify theparticular DVD/CD inserted and copied to the storage device 230. Forexample, a checksum may be calculated for a known unique portion of theDVD/CD and compared with a CD/DVD checksum database (e.g., maintained atthe NOC 180 or other server). Similarly, the serial number may be readfrom the DVD/CD and compared with a database of DVD/CD serial numbers.Additional DVD/CD identification techniques which may be utilized inaccordance with the principles of the invention are disclosed inco-pending applications entitled SYSTEM AND METHOD FOR SCALING A VIDEOSIGNAL, Ser. No. 09/632,458, filed Aug. 4, 2000 which is assigned to theassignee of the present application and which is incorporated herein byreference.

In one embodiment of the system, the storage device 230 (e.g., a harddrive) is preferably large enough to store hundreds of hours of videoand/or audio content, as well as a variety of other digital informationsuch as telephone voice messages, computer programs/data . . . etc. Thecurrent recommended size for the storage device 230 is at least 80gigabytes, however the particular size of the storage device is notpertinent to the underlying principles of the invention.

One or more RF transmitters 430 are also provided in one embodiment ofthe home media server 110. The transmitter 430 (as well as the LAN 390,if one is installed) allows the home media server 110 to simultaneouslytransmit multimedia content and other types of data to the various mediadevices 150–151, 160–166 over the home media network 140 (e.g., at leastone MPEG-2 video stream and several audio streams).

In one embodiment, the wireless transmission system is capable ofhandling isochronous multimedia traffic reliably (i.e., without jitter)in a wide range of residential settings. For example, the system must becapable of working in the presence of common residential RF interferencesuch as microwave ovens and cell phones. In one embodiment, theserequirements are met through the proper choice of modulation and coding.More specifically, one embodiment employs a modified version of IEEE802.11b adapted to operate in a real time environment (e.g., usingReed-Solomon forward error-correction and antenna diversity withcircular polarization).

FIG. 5 illustrates a home media server 110 communicating simultaneouslywith a video node 520 and an audio node 522. Certain multimedia nodes,such as node 522, include an LCD 524 (or other type of display) fordisplaying information about the multimedia content stored on home mediaserver 110 (e.g., CD and movie titles, CD tracks . . . etc). In oneembodiment, the audio node 522 may be electrically coupled to anamplifier for amplifying the transmitted audio signal into a pair ofspeakers. As described above, in one embodiment, the individualmultimedia nodes also include local storage devices (not shown) forlocally caching recently used multimedia content and data.

Data/commands sent from remote control devices 530, 532 are transmittedthrough the nodes 520, 522 to the home/media server 110 as indicated. Inone embodiment, the remote control devices include an LCD (not shown) orother type of display for displaying information about the multimediacontent stored on home media server 110 (i.e., instead of, or inaddition to, the multimedia node 522 display 524).

As described briefly above, using the foregoing system, all of a user'sdata, music and video may be stored in a single location (i.e., homemedia server 110) and accessed from anywhere in the house (e.g., stereonode 522) or the car (e.g., via an automotive multimedia node 164 asindicated in FIG. 1). Moreover, if the home media server 110 isconnected to the Internet through, for example, a persistent DSLconnection 360, the user can access all of the stored content fromvarious other locations across the globe (e.g., a summer home or a hotelwhile away on business). One embodiment of the system provides a secure,encrypted data stream when content/data is requested from the home mediaserver 110 in this manner, thereby protecting the user's privacy as wellas the copyrights to the underlying multimedia content.

If a multimedia node is employed in a user's automobile as describedabove, when the automobile is parked within range of the media server'sRF transmissions, music or other audio/video content may be transmittedand stored on a storage device within the automobile, even when it isnot running. The specific type of audio/video content to be transmittedat a particular time during the day or week may be variable, based onusers' preferences. For example, the home media server 110 may beconfigured to transmit up-to-date traffic reports during the morning andevening hours before the user leaves home/work. The user will thenreceive an instant traffic report as soon as he starts his car in themorning and/or evening, followed by music or video selected based on theuser's preferences. Various additional aspects of this feature are setforth in the co-pending applications incorporated by reference above.

In an embodiment which includes a CD-ReWritable (“CD-RW”) drive, userscan produce custom CDs using the audio content stored on the massstorage device 230 or downloaded from the Internet. The CD-RW drive mayutilize serial copy management techniques to produce CDs which cannotthemselves be copied, thereby protecting the copyrights to theunderlying works. Moreover, in one embodiment, the system will determinewhether the content owner or content creator allows copying of themultimedia content. For example, certain content creators/owners mayallow a copy to be stored on the home media server mass storage device230, but may not allow copying to a portable disk (e.g., a CD-RW disk).In one embodiment, content stored on the mass storage device 230 may belabeled as non-copyable by setting one or more “copy” bits to aparticular value (e.g., in a bit field associated with the content).

In one embodiment of the system, a portable music/video player node maybe configured to retrieve multimedia content directly from the homemedia server 110 or via an adapter module. Portable MP3 players such asthe Rio™ from Diamond Multimedia™, for example, may be used to store andplay back digital audio content transmitted from the home media serverin a flash memory module. In one particular embodiment, the CD-RW drive420 is capable of accepting the ⅓ inch “mini CD” format. Accordingly,users may produce unique mini CDs (e.g., using tracks stored on the massstorage device 230), for playback on MP-3 players capable of playingmini-CDs.

MPEG conversion logic is configured on one embodiment of the home mediaserver 110 (e.g., in software, hardware or any combination thereof),allowing the home media server 110 to convert from one MPEG format toanother. Using the MPEG conversion logic, video content stored in MPEG-2format may be converted to MPEG-1 format and recorded on a “video CD” (acompact disc format used to hold full-motion MPEG-1 video). As such,users may store video on a portable medium, even on a home media server110 which is not equipped with a recordable DVD drive.

In another embodiment of the system, home appliances (e.g., therefrigerator, the toaster, the air conditioner) and other home systems(e.g., security, air conditioning) are all provided with RF transmissiondevices to communicate with the home media server 110. Each device mayalso be configured with it's own internal network address and/orInternet address. Users may then access information pertaining to thesedevices and/or control these devices from any room in the home or overthe Internet. In one particular embodiment, the user's automobile isoutfitted with an RF transmitter and a network address. Thus, in thisembodiment, the automobile is capable of reporting maintenanceinformation to the user via the home media server 110 (e.g., low brakepads, oil change needed . . . etc).

In one embodiment, a user may publish a home Web page containingup-to-date information on each home appliance or other network device.Accordingly, users of this embodiment are able to monitor and controlhome appliances and systems from anywhere in the world. In oneembodiment this includes the ability to select broadcast listings anddirect the home media server 110 to make recordings (e.g., based ondate/time or broadcast ID code). In addition, in one embodiment, usersmay connect remotely to the home media server to review email and/orvoicemail listings (which may be displayed to the user in the form of asingle, generic “message” box). Voicemail messages may be streamed tothe remote user's location over the Internet or other network.

Telephony Embodiments

As illustrated in FIG. 7 a, in one embodiment, home telephone devices166 may also be configured to run through the home media server 110. Inthis embodiment, incoming faxes and voicemail are stored on the homemedia storage device 230 and may be accessed from any room in the house(e.g. from telephone devices 166, personal computers 160, PDAs 165,and/or video systems 151/171). In addition, phone number and addressinformation may be stored in a contact database 740 on the home mediaserver 110 and accessed through the various telephone devices 166 (orother home media devices). Offloading user contact data from thetelephone devices 166 in this manner allows telephone devices 166 to bemanufactured will less memory and less processing power, furtherdecreasing costs to the end user.

As illustrated in FIG. 7 a, in one embodiment, the home media server 110includes a telephone connection module 730 which coordinates betweenstandard telephony calls placed/received over the public switchedtelephone network (“PSTN”) 170 and calls placed over the Internet 120using IP telephony protocols. The telephone connection module 730 willautomatically route incoming calls from both sources to the same set ofhome telephone devices 166 (or other home media devices such as theuser's personal computer 160).

Users may specify whether a particular outgoing call should be placedover the Internet 120 (e.g., to an IP telephony device 710) or over thePSTN 170 (e.g., to a standard telephone device 720). In one embodiment,the telephone connection module 730 analyzes each outgoing telephoneconnection request to determine whether the call should be routedthrough the Internet 120 or through the PSTN 170. The telephoneconnection module 370 may factor in various types of connection data tomake telephone connection determinations. For example, a user mayspecify certain contacts within the contact database 740 for which IPtelephone connections should be used and certain contacts for whichstandard telephone connections should be used. In one embodiment, thetelephone connection module 730 will select a particular connectionbased on whether it is the least expensive option for the user (e.g.,for contacts with both IP and standard telephone capabilities). Anothervariable which may be factored into the selection process is theconnection throughput available to the user over the Internet. Morespecifically, the telephone connection module 730 may initially test theconnection throughput (including the availability of a reliableconnection) over the Internet 120 and place an IP telephony call only ifsome minimum level of throughout/reliability is available. Any of theforegoing variables, alone or in combination, may be evaluated by thetelephony connection module 730 to select and appropriate telephoneconnection.

One embodiment of a protocol architecture for supporting IP telephonyand related communication functions on the home media server 110 isillustrated in FIG. 2 b. Various telecommunication standards aresupported by this embodiment, including the H.248 media gateway controlstandard (“MEGACO”) standard; the ITU-T H.323 and session initiationprotocol (“SIP”) standards for multimedia videoconferencing onpacket-switched networks; the Realtime Control Protocol (“RTCP”)standard—a companion protocol to the realtime transport protocol (“RTP”)(described above) used to maintain a minimum QoS; and the ITU T.120standard for realtime data conferencing (sharing data among multipleusers).

In addition, various audio and video codecs are supported by theillustrated embodiment, including G.711, G.723, and G.729 (for audio);and H.261 and H.261 (for video). As illustrated, each of these codecsmay be executed above RTP, an IP protocol that supports realtimetransmission of voice and video. Each of the foregoing IP communicationprotocols may be utilized by media server 110 application programsthrough the programming interfaces of the media toolkit 260.

It should be noted that, although the embodiment illustrated in FIG. 7 bincludes a specific set of communication protocols, various othercommunication protocols may be implemented within the home media server110 while still complying with the underlying principles of theinvention. For example, in one embodiment, new protocol stacks (bothproprietary and industry-standard) may be automatically transmitted fromthe NOC 180 and installed on the home media server 110 as they becomeavailable.

FIG. 7 c illustrates a signal diagram representing the set up andtermination of a telephony connection according to one embodiment of thehome media server 110. At 750, the home media server transmits a callconnection “invitation” on behalf of User 1 to User 2. Once User 2accepts the call, a “success” message is transmitted back to the homemedia server 110 at 751. In response, the home media server 110acknowledges receipt of the “success” message at 752 and allocates amedia stream 760 to support bi-direction audio communication betweenUser 1 and User 2. To tear down the media stream 760, one of the usersmust hang up the phone, resulting in a “bye” message 753, 754 followedby a termination acknowledgement (“success”) from both sides 755, 756.

Broadcast and Packet-Switched Channel Coordination

In one embodiment illustrated in FIG. 8 b, content providers 830 maytransmit content to home media server 110 over Internet channels 120 aswell as standard broadcast channels 170. The transmitted content maythen be cached locally in a content database 850. When a usersubsequently requests the same content to be played back on an audiodevice or video device 171 (e.g., via media node 151), it will bereplayed to the user directly from the local content database ratherthan over a broadcast channel. In one embodiment, a content selectionmodule 840 will determine whether to retrieve the content through abroadcast channel or directly from the content database 850.

In one embodiment, the home media server 110 and/or the content provider830 will monitor the preferences of each user in the media server 110household to determine the content which will be requested duringcertain times of the day, and/or days of the week. For example, if auser watches “the Simpsons®” at 6:00 PM, the home media server 110and/or content provider 830 will record this behavior. The contentprovider 830 may then transmit the latest Simpsons episode to the homemedia server 110 over the Internet 120 before its scheduled broadcast at6:00 PM (e.g., during periods of the day or evening when traffic on theInternet 120 is low). Various other pre-recorded broadcast programs maybe transmitted to the home media server 110 over the Internet ratherthan over a dedicated broadcast channel, thereby freeing up asubstantial amount of inefficiently used broadcast network bandwidth(pre-recorded content currently represents the vast majority of allbroadcast content).

As indicated in FIG. 8 b, this embodiment of the home media server 100will provide a seamless mechanism for transitioning from standarddigital broadcast channels (e.g., MPEG-2 channels) and analog broadcastchannels to packet switched isochronous channels (for live broadcasts)and asynchronous channels (for pre-recorded broadcasts). Initially, thebreakdown between broadcast and packet switched channels (e.g., TCP/IP)will be similar to that illustrated in column 804, with analog broadcast803 taking up approximately half of the available bandwidth, with theother half split between digital broadcast 802 and packet switchedchannels 800. As content providers begin to switch over to apacket-switched transmission scheme, this breakdown will change asindicated at bandwidth allocation 805, with packet-switched channelstaking up approximately half of the available bandwidth and theremainder split between analog broadcast 812 and digital broadcast 811.

Finally, when standard analog and digital broadcast channels have beenphased out completely, all content will be delivered over packetswitched channels as indicated in bandwidth allocation 806. Many packetswitched protocols support both asynchronous and isochronous datatransmissions. Accordingly, several isochronous channels may be providedto support live, real time events (e.g., sports events, the eveningnews, . . . etc) and the remainder of the bandwidth may be allocated topre-recorded events (e.g., sitcoms, game shown, talk shows, . . . etc).These pre-recorded events/shown may be asynchronously downloaded to thehome media server 110 at any time of the day or evening (e.g., duringperiods of low network activity), resulting in a much more efficientallocation of available transmission bandwidth.

Because embodiments of the home media server 110 described above arecapable of receiving, storing and coordinating content transmitted fromboth packet switched channels (e.g., TCP/IP) and standard broadcastchannels, the transition illustrated in FIG. 8 b may be accomplishedseamlessly, while at the same time improving the overall quality ofservice to the end user.

In one embodiment, several concurrent, real-time multimedia streams aresupported by the home media server 110 through buffering and/or diskstorage techniques as illustrated in FIG. 8 c. More specifically,incoming multimedia content 860–862 from several different channels(e.g., standard broadcast channels 860, 861 and packet switched channels862, 863) may be buffered in a set of input buffers 870–873 and outputbuffers 890–893 on the home media server 110. The input buffers andoutput buffers 870–873 and 890–893, respectively, may be portions ofmemory allocated within the main memory 201 (see FIG. 2). Alternatively,or in addition, the input/output buffers may be configured on thecommunication modules 240–245, within the system ASIC 210, and/or asseparate modules on the home media server 110 motherboard.

In one embodiment, the mass storage device 230 reads the multimedia datafrom each of the input buffers 870–873 and writes the data to a set ofmultimedia tracks 880–883. The multimedia data is subsequently read fromeach of the multimedia tracks 880–883 to a set of output buffers890–893, from which it is transmitted to one or more of the destinationmultimedia devices 150, 151, 163. Buffering data streams on the homemedia server 110 in this manner provides significant cost efficienciesfor the entire system. Specifically, because the multimedia data isbuffered at a single distribution point within the home media network140 (i.e., the home media server 110), no buffering is required at eachof the individual multimedia devices 150, 151 and 160–166, therebysignificantly reducing the cost and complexity of these devices.

The mass storage device 230 of this embodiment has enough bandwidth inits read/write operations to support several multimedia streams at once.Although the seek time of the mass storage device 230 (i.e., the timerequired to jump from one track to another) may be relatively low (i.e.,relative to its read/write bandwidth), buffering of the multimedia datain input and output buffers as described ensures that the variousstreams will be provided to their respective multimedia devices 150, 151and 160–166 without interruption.

In one embodiment, each of the multimedia streams 860–863 operateindependently of one another. For example, if playback of a particularmultimedia stream is paused, e.g., from multimedia device 150, the homemedia server 110 will interrupt transmission of multimedia content fromthe output buffer 893 associated with that device 150, and will beginstoring subsequent, incoming multimedia content on the mass storagedevice 230 (e.g., within the media track 883). However, this will notaffect playback at any of the other multimedia devices on the network140. When playback is resumed, the content will be streamed from theoutput buffer 893 (and media track 883) from the same point at which iswas paused (thereby providing real time, time-shifting functionality forlive, real-time broadcasts).

In one embodiment, an indexing module (not shown) will generate acontent index for users of the home media server 110 which will provideusers with a comprehensive, seamless listing of multimedia content fromthe Internet (e.g., streaming media content), broadcast channels(including live and pre-recorded broadcast channels),asynchronous/isochronous multimedia channels, and/or various other mediasources/channels. Accordingly, this embodiment will provide users withthe ability to navigate through and select content from a virtuallyunlimited number of different multimedia sources and channels having anyknowledge of the underlying protocols and communication infrastructuresupporting the sources/channels.

Intelligent Buffering and Bandwidth Allocation

Virtually all communication channels are bandwidth-limited in somemanner, due to the physical limitations of the underlying transmissionmedium and/or the signaling limitations of the channel (e.g., thechannel's allocated frequency spectrum). For example, a 100 Base-TEthernet network is capable of providing a total data throughput of 100Mbps, which is shared by all nodes (e.g., computers) on the network.Similarly, multimedia devices 150, 151; 160–166 communicating on thehome media network 140 described above all share some maximum networksignaling rate (e.g., if a standard IEEE 802.11b network is employed thesignaling rate is 11 Mbps).

Because bandwidth is shared on these systems, overall networkperformance may degrade as a result of nodes performing concurrent,high-bandwidth network transactions. This can be particularlyproblematic on networks such as the home media network 140 which supportreal-time multimedia services. More specifically, high quality audio andvideo signals transmitted from the home media server 110 to multimedianodes 150, 151, require some minimum level of network bandwidth to berendered properly (i.e., without distortion or interruption).Accordingly, one embodiment of the invention includes intelligentbuffering and bandwidth allocation techniques to ensure smooth playbackof the multimedia signal at each of the multimedia nodes 150, 151.

FIG. 9 a illustrates a histogram of normalized bitrate data for aparticular multimedia stream (e.g., a DVD) over a period of 120 seconds.Curve 902 shows the bitrate averaged at one second intervals whereascurve 904 shows the bitrate averaged at 10 second intervals. The averagebitrate over the entire 120 second interval is represented by curve 906(approximately 4.7 Mbps). Accordingly, based on the histogram data, theeffective bitrate required to properly render the video stream atmultimedia node 150 fluctuates significantly—from a maximum throughputof 7.143 Mbps to a minimum throughput of 1.657 Mbps.

One embodiment of the home media server 110, illustrated in FIG. 9 b,uses this type of bitrate data to allocate bandwidth and bufferinglevels for multimedia playback. Accordingly, when a user selects audioor video content (e.g., a DVD) to be transmitted to a multimedia node(e.g., video node 151), the home media server 110 initially retrieves abitrate template 930 (e.g., a histogram of bitrate data) associated withthe requested content from a bitrate database 935. The database 935 maybe maintained locally on the home media server 110 and/or, asillustrated, may be maintained on a remote database server 940 (e.g.,maintained at the NOC 180). In either case, the database 935 may becontinually updated as new multimedia content becomes available. Oncethe bitrate template 930 is initially downloaded from the remote server940, a copy may be stored locally on the home media server 110 forfuture reference. In one particular embodiment, the bitrate template 930may be transmitted along with the underlying multimedia signal (e.g.,embedded within the MPEG-2 stream read from a DVD or broadcast over acable channel), rather than maintained in a separate database 935.

If the bitrate template 930 is maintained in a database 935, varioustypes of identification data 925 may be used to identify the correcttemplate for the multimedia content being played including, but notlimited to, those set forth in co-pending application entitled SYSTEMAND METHOD FOR SCALING A VIDEO SIGNAL, Ser. No. 09/632,458 (incorporatedby reference above). For example, a checksum may be calculated for aknown unique portion of the multimedia content and compared withchecksums stored in the database 935. Similarly, if the content isstored on a CD or DVD, the CD/DVD serial number (or other embeddedidentification code) may be used to perform a database 935 query.Various other identification techniques may be used to identify themultimedia content while still complying with the underlying principlesof the invention. Once the correct bitrate template 930 is identified,it is transmitted from the remote server 940 to the home media server110. Alternatively, if the data is stored locally on the home mediaserver 110, it is retrieved directly from the home media server'sstorage device 230.

In one embodiment, an allocation module 950 running on the home mediaserver 110 analyzes the bitrate template 930 to establish an efficientbandwidth allocation and/or buffering policy for transmitting themultimedia stream(s). The allocation module 950 acts as a data“throttle,” increasing or decreasing the data throughput from the homemedia server 110 to the multimedia nodes 150–151 as necessary to meetthe bitrate requirements of each multimedia stream (e.g., throughcontrol of the home media server output buffers 910–912 and/or RFtransceiver 915). The goals of one embodiment of the system are (1) toensure that the underlying multimedia content is reproduced at themultimedia node 151 without interruption/jitter; (2) to minimize thememory requirements at the multimedia node 151; (3) to minimize theplayback delay experienced by the end user; and/or (4) to minimize thebandwidth required to accurately reproduce the multimedia content at thenode 151. Any of these goals, alone or in combination, may be factoredinto the allocation module's 150's bandwidth/buffering allocationpolicy.

In one embodiment, the allocation module 950 analyzes the bitratetemplate 930 to ensure that the amount of multimedia content in eachmultimedia node buffer—e.g., buffer 920 of multimedia node 151—issufficient to handle upcoming spikes. For example, given the bitratetemplate data set forth in FIG. 9 c, the multimedia node buffer 920 musthave sufficient multimedia data (i.e., an adequate number of bits) tohandle the bitrate spike between 3 seconds and 4 seconds (i.e., 6.2Mbps). As such, by analyzing the bitrate template 930 as a whole, theallocation module 950 may increase the allocated data throughput between1 second and 3 seconds to sufficiently fill the buffer 920 by the timethe bitrate spike arrives (i.e., at 3 seconds). Thus, the number of bitsconsumed during the spike (6.2 Meg) must be less than or equal to thenumber of bits in the buffer at start of spike period (3 seconds) minusthe bits added to buffer during spike period (i.e., the per-secondbitrate). Otherwise, playback of the multimedia stream will stall due toan underrun condition (i.e., a lack of multimedia data at the multimedianode 151).

The allocation module 950 may also factor in bitrate templates 930 ofother multimedia streams when making allocation decisions for a givenstream. For example, when making allocation decisions for the DVD streamin FIG. 9 b (which passes through buffers 910 and 920), the allocationmodule 950 may evaluate the bitrate requirements of the other twostreams (i.e., the MP3 stream and Cable stream passing through buffers911, 921 and 912, 922, respectively). As such, if one particular streamrequires a significant amount of throughput at a given point in time,the allocation module will take anticipatory steps to ensure thatsufficient multimedia data will be available. For example, in FIG. 9 d,the bitrate data illustrated in timeline 960 indicates a severe bitratespike between 40 and 60 seconds. The spike is severe enough that inorder to properly render the multimedia stream associated with it, theallocation module 960 may need to fill the multimedia node buffer by thetime the spike arrives (i.e., at 40 seconds) and also divert asignificant amount of the system bandwidth to the stream to avoid anunderrun condition. As such, in anticipation of the spike, theallocation module 950 may attempt to fill all the buffers on the systemincluding the buffers of the other two streams (associated with thebitrate data shown in timelines 961 and 962) so that by the time thespike arrives, it can divert bandwidth from these streams to the streamwith the bitrate spike. It should be noted that the foregoingbandwidth/buffer allocation example was for the purpose of illustrationonly and should not be read to limit the scope of the invention.

In one embodiment, the when a new playback stream is requested at amultimedia node 151, the allocation module 950 will initially allocateall system bandwidth available at that time to the stream. It willsubsequently decrease the bandwidth (i.e., the “throttle”) only whenother multimedia nodes require additional bandwidth and/or when thebuffer at the multimedia node 151 fills up. Accordingly, one goal ofthis embodiment is to keep the buffer at the multimedia node 151 filledto its limit at all times (e.g., so that it will be ready for upcomingspikes).

Playback of the multimedia data from the multimedia node buffer 920 maystart at different times, depending on the histogram data associatedwith the content. For example, the bitrate for certain high definitiontelevision broadcasts or DVDs such as the one illustrated in FIG. 9 e(which shows bitrate data for the movie “South Park”) may start off at arelatively high value (i.e., approximately 8.6 Mbps). As such, it may benecessary to accumulate a certain amount of multimedia data in themultimedia node buffer 920 before initiating playback. The table 970illustrated in FIG. 9 e shows the amount of data that must be bufferedbefore playback (column 972), given a particular multimedia streambitrate (column 971). The table 970 also indicates the amount of timewhich a user must wait before playback begins (column 973). As indicatedin the bottom row of the table 970, at a bitrate of 9 Mbps, no bufferingof the multimedia signal is required. In one embodiment, playback isinitiated by the allocation module 950 (as indicated by the controlsignal 952 in FIG. 9 b).

In contrast to multimedia stream described by the histogram of FIG. 9 e,the multimedia stream described by the histogram illustrated in FIG. 9 f(the DVD for the movie “Species 1”) requires significantly lessbuffering before playback. More particularly, as indicated in table 975,no pre-playback buffering is required at a bitrate of 5 Mbps. As such,if 5 Mbps or more system bandwidth is available, playback may begin assoon as the stream reaches the multimedia node 151. In addition, in oneembodiment, the allocation module 950 may initially allocate morebandwidth than required to fill the multimedia node buffer 920 duringplayback (e.g., so that more system bandwidth may be allocated elsewhereonce the buffer is full).

In one embodiment, when a new playback stream is requested from amultimedia node 151, the allocation module 950 may initially allocatejust enough system bandwidth to meet some minimum playback delaythreshold. Thus, if this minimum delay threshold is set at 4.5 seconds,the allocation module 650 must allocate at least 7 Mbps to reproduce themultimedia content represented by the histogram of FIG. 9 e (see Table970, 5^(th) row). By contrast, to meet the minimum delay threshold forthe multimedia content represented by the histogram of FIG. 9 f, theallocation module 950 may allocate a bitrate of less than 5 Mbps (seeTable 975). After the initial bandwidth allocation, the allocationmodule may use the histogram data to ensure that the amount ofmultimedia content in the multimedia node buffer 920 is sufficient tohandle upcoming spikes in bitrate (as described above).

Various additional bitrate histograms and related bandwidth allocationtables are set forth in FIGS. 9 g through 9 m. It should be noted,however, that these histograms are provided merely for the purpose ofillustration, and should not be read to limit the scope of the presentinvention. Moreover, although the examples set forth above focus on thereproduction of DVD movies at multimedia nodes on a home media network,it should be noted that the underlying principles of the invention maybe implemented on virtually any system in which time-based data istransmitted over a bandwidth-limited network. Accordingly, the scope andspirit of the present invention should be judged in terms of the claimsset forth below.

Conditional Access and Encryption

One embodiment of a cable broadcast module for receiving one or morebroadcast channels is illustrated in FIG. 10. This embodiment includestwo tuners 1010, 1011 for concurrently tuning to two separate broadcastchannels (e.g., to be viewed at two separate multimedia nodes on thehome media network 140). Additional tuners may be included within thecable broadcast module as necessary (i.e., for receiving additionalbroadcast channels).

In one embodiment of the system, a smartcard 330 (see also FIG. 3)inserted into one or more of the media modules 320–325 may be programmedwith a specific set of access rights defining the particular channels towhich users are permitted access. The smartcard 330 may work inconjunction with a conditional access module 1030, which will preventcertain channels/content from being transmitted on the system which theuser does not have the right to receive. In one embodiment, differentmembers of a family may be assigned smartcards 330 with different accessprivileges. For example, children may be assigned smartcards 330 whichlimit the channels and/or Internet sites which they are permitted toaccess.

In the illustrated embodiment, the signal outputs from each of thetuners 1010, 1011 are fed into separate audio and videoanalog-to-digital decoder modules 1024, 1026 (audio) and 1023, 1025(video), respectively. In one embodiment, the signals are thenre-encoded with a compression codec. For example, in the embodimentshown in FIG. 9, MPEG-2 encoder modules 1031, 1033 encode theaudio/video signals into MPEG-2 format (i.e., which may subsequently bedecoded by any multimedia devices 150, 151 on the home media network140). It should be noted that various other digital compression codecsmay be used (e.g., AC-3, MPEG-1, . . . etc) while still complying withthe underlying principles of the invention.

In addition, in one embodiment, before being transmitted onto the systembus 220, the encoded multimedia signals are encrypted by an encryptionmodule 1060 using a unique encryption key 1061. As such, in thisembodiment, all multimedia data stored on the home media server 110(i.e., on the mass storage device 230) and transmitted across the homemedia network 140 are encrypted, thereby preventing copying and playbackby any unauthorized devices listening in one the network 140 or on thesystem bus 220. Moreover, authorized multimedia devices 150, 151,160–166 (i.e., those certified by the NOC 180) communicating on the homemedia network 140 are configured with the same unique encryption key1061, thereby allowing them to properly decode the multimedia datatransmitted from the home media server 110.

In one embodiment, the encryption key 1061 is transmitted securely tothe home media server 110 and media devices 150, 151; 160–166 throughthe conditional access system. Alternatively, or in addition, the key1061 may be transmitted using conventional secure communicationtechniques such as Secure Sockets Layer (“SSL”) or Data EncryptionStandard (“DES”).

One embodiment of a satellite module for connecting the home mediaserver 110 over a satellite channel is illustrated in FIG. 11, includingmultiple tuners 1110, 1111; forward error correction modules 1130, 1140;a smartcard 1120 and associated conditional access module 1150; and anencryption module 960 for encrypting content transmitted onto the systembus 220 (and across the home media network 140). The different decoding,encryption, and authentication features described above with respect tothe cable broadcast module illustrated in FIG. 9 are also configuredinto one embodiment of the satellite module.

One embodiment of a cable modem for connecting the home media server 110over a cable channel is illustrated in FIG. 12. The cable modem includesa modulator unit 910, a Data Over Cable Service Interface Specification(“DOCSIS”) module 920, and a data pump/PCI interface unit 930. Variousdifferent types of cable modems may be coupled to the home media server110 while still complying with the underlying principles of theinvention.

In addition to the encryption system described above, one embodiment ofthe system provides copyright protection through the use of copyrighttags associated with content stored on the home media server 110. Thetags of this embodiment identify which types of content/information maybe copied or streamed to which multimedia devices. As illustrated inFIG. 13, tags may be set to a variety of copy/playback levels,including, for example, a tag 1311 indicating no restrictions on copyingfor audio file 1310; a tag 1321 allowing only a single serial copy(e.g., to a CD-RW disk) with no copying to personal computers for audiofile 1320; a tag 1331 permitting playback to local TVs (i.e., within thehome media network 140) for video file 1330; and a tag 1341 allowing24-hour playback to local TVs only (e.g., similar to a pay-per-viewfeature) for video file 1340. Accordingly, this embodiment provides anadditional mechanism for providing secure storage and transmission ofcopyrightable material. In one particular embodiment, content providers(e.g., cable and satellite providers) are provided with the ability toprogram tags into their content as necessary.

A System and Method for Multimedia Stream Conversion Introduction

In one embodiment of the invention, the Media Server 110 or othermultimedia receiver system (e.g., a standard cable/satellite receiver)is configured to process and render audio, video and sub-picture contentfrom a DVD using a transport stream decoder. Certain relevant featuresof MPEG compression, MPEG transport and program streams, andsupplemental DVD media types and metadata will now be provided followedby a detailed description of embodiments of the invention.

MPEG Compression

As illustrated in FIG. 14, an MPEG2 “elementary stream” is the rawMPEG-compressed audio/video data generated by an MPEG-2 encoder 1410.The MPEG-2 encoder exploits both spatial redundancies and temporalredundancies in the video images provided at its input. Spatialredundancies are exploited by using field-based Discrete CosineTransform (“DCT”) coding of 8×8 pixel blocks followed by quantization,zigzag scan, and variable length coding of runs of zero-quantizedindices and amplitudes of those indices. Quantization scaling factorsand quantization matrices are used to effectively remove the DCTcoefficients containing perceptually irrelevant information, therebyincreasing the MPEG-2 coding efficiency. Temporal redundancies areexploited by inter-coding, or transmitting only the differences betweenpictures. A “macroblock” is the graphical unit used for temporalcalculations and is comprised of four 8×8 luminance (grayscale) (Y)components and two 8×8 chromatic (color) components (one for Cb and onefor Cr).

In MPEG-2, three different types of “frames” or “pictures” are used toexploit the spatial and temporal redundancies of the sequence of videoframes while minimizing error propagation:

“I frames” or “I pictures” are individually DCT-coded pictures that needno additional information for decoding. As such, they require asignificant amount of data compared to other frame types, and thereforethey are not transmitted any more frequently than necessary. I framesconsist primarily of DCT transform coefficients and have no vectors(i.e., only spatial redundancies are exploited to generate I frames).

“P frames” or “P pictures” are forward-predicted from an earlier frame,which could either be an I frame or another P frame. P frame dataconsists of vectors describing where, in the previous frame, eachmacroblock should be taken from, and not of transform coefficients thatdescribe the correction or difference data that must be added to thatmacroblock. P frames require roughly half the data of an I frame.

“B frames” or “B pictures” are bi-directionally predicted from earlieror later I or P frames. B-frame data consists of vectors describingwhere in earlier or later frames data should be taken from. It alsocontains the transform coefficients that provide the correction. Becausebidirectional prediction is so effective, the correction data areminimal and this helps the B frame to typically require one quarter thedata of an I frame.

Packetized Elementary Streams

Audio and video PES packetizers 1420 and 1422, respectively, split eachelementary stream into variable-length packets to form a PacketizedElementary Stream (“PES”). Each PES packet includes a header whichcontains information about the encoding process required by the MPEGdecoder (e.g., such as an indication as to whether the packet containsaudio or video content). In addition, in order to keep the audio andvideo locked together, time stamps are periodically incorporated withinPES packets. A PES time stamp is a 33-bit number that is a sample of acounter driven by a 90-kHz clock.

Transport Streams and Program Streams

To create a transport stream, a transport stream multiplexer 1431 splitsPES packets into fixed-length 188 byte transport stream packets with aminimum 4-byte header and a maximum 184-byte payload. In addition, asignificant amount of metadata is multiplexed in the transport stream.For example, during the formation of a transport stream, programspecific information (“PSI”) packets may be inserted containing tablesrequired to demultiplex the transport stream and conditional access(“CA”) tables may be inserted containing information needed by thereceiver to decrypt certain programs (e.g., subscription-based programssuch as HBO). In order to keep the bitrate constant at the receiver,null packets containing “stuffing” bytes may also be inserted to fillthe intervals between information-bearing packets.

The different programs multiplexed within the transport stream may haveoriginated in different places that are not necessarily synchronized. Assuch, the transport stream must provide a separate synchronizationmechanism for each transmitted program. To this end, 33-bit sequentialprogram clock reference (“PCR”) values are inserted into the optionalheader fields of certain transport stream packets. The PCR valuesrecreate a stable reference clock that can be divided down to create atime line at the decoder, so that the time stamps for the elementarystreams in each transport stream become meaningful. Consequently, onedefinition of a “program” within a transport stream is a set ofelementary streams sharing the same timing reference.

As described in ISO/IEC 13818-1, the program stream was designed for usein relatively error-free environments and is particularly suitable forapplications which may involve software processing of system information(e.g., such as DVD applications). Program stream packets may generallybe of a variable length. The DVD specification calls for program streampackets of 2048 bytes in length.

The program stream generated by a program stream MUX 1430 is a multiplexof PES packets that, unlike the transport stream, were all encoded usinga common time reference (i.e., the same master clock or system timeclock). The timestamps embedded within program stream packets are knownas system clock reference (“SCR”) values. Like the PCR values usedwithin the transport stream, SCR values are 33 bits in length. However,as will be described in detail below, as a user navigates throughmultimedia content stored on a DVD the SCR values for program packetsread from the DVD are not necessarily consecutive.

A DVD program stream may depart from the MPEG specification in thatmultiple “programs” (in the sense of multiple sets of elementarystreams, each set sharing a timing reference independent of the othersets) are allowed as interleaved units. If present, these extra streamsshare the same stream ID, making them completely unplayable by anystrictly conforming program stream decoder or transport stream decoder,without the repacketizing techniques described below.

Additional DVD Media Types and Metadata

As mentioned above, in addition to the raw program stream packets storedon a DVD, the DVD specification defines additional media types and datastructures that are not part of the MPEG specification. In particular,non-MPEG audio formats (e.g., AC-3, PCM, DTS, . . . etc), subpicturedata, and navigation packs are packaged in private data stream PESpackets whose payload format is outside of the MPEG specification. EachDVD-Video disc also includes a main directory known as the VIDEO_TSdirectory which contains two types of files: “.IFO” files and “.VOB”files (i.e., indicating the file extension). During playback, thesefiles are sorted by a DVD video player to form video “title” sets, whichare groupings of all files necessary to play a particular DVD video“title” (e.g., a movie). Each video title set is composed of one .IFOfile and one or more .VOB files.

The .VOB file(s) which contain the underlying multimedia data and arereferred to as the video object set (VOBS). The location and format ofthe multimedia data stored in the VOBS is defined by the associated .IFOfile. More specifically, the .IFO file contains the navigational datastructures and a processor-independent interpreted language whichspecifies how the data structures are arranged.

The data structures themselves are composed of various objects known as“program chain objects,” “program objects,” and “cell objects.” Programchain objects link related program objects (e.g., particular scenes)within a title and thereby govern the playback of the program objects.Programs, in turn, point to one or more cells for presentation. Aparticular title may contain only a single program chain. More complextitles may contain two or more program chains to allow random access toa variety of programs. As will be described in greater detail below, thecells allocated within a single program or between different programsmay be connected seamlessly (i.e., using a common time base) ornon-seamlessly (i.e., using different, unrelated time bases).

As indicated in FIG. 15, each program object 1510 in a program chain iscomposed of “cell objects” 1520 which identify to a DVD player theportion of the multimedia data to decode from one of the .VOB files. Thedata structures within cell objects 1520 are defined in the .IFO file.Each cell object directs the DVD player to begin playback at a specificlocation in the .VOB file referred to as a video object unit (VOBU) 230.A VOBU is a container object that includes a navigation header 1540,referred to as a navigation pack, audio and video program stream packets1541 and 1542, respectively, and packets containing sub-picture data1543. The navigation header 1540 includes the presentation controlinformation (PCI) and data search information (DSI) mentioned above.

Embodiments of the Invention

FIG. 16 a generally illustrates one embodiment of a multimedia systemconfigured to process and render audio, video and sub-picture contentfrom a DVD 1600 using a transport stream decoder 1620. In response touser input selections 1606 and navigation data stored on the DVD 1600(e.g., contained in the .IFO files), a DVD navigation module 1605navigates a play path through the DVD 1600 via DVD hardware 1602. TheDVD hardware 1602 is comprised of well known optical and mechanicalcomponents required to read content from a DVD. The content selected bythe DVD navigation module 1605 is transmitted to a content conversionmodule 1610 which translates the DVD content into a format which thetransport stream decoder 1620 can process and render on a display 1630(e.g., a standard television display, a computer display, . . . etc). Inaddition, in one embodiment, a resynchronization signal 1616 isgenerated by the content conversion module 1610 to cause the transportstream decoder 1620 to resynchronize with the converted DVD content 1612under certain conditions, described in detail below.

In one embodiment, the DVD navigation module 1605 and content conversionmodule 1610 are implemented in software executed by a central processingunit (“CPU”) 1612 configured within a PVR system or standard multimediareceiver system. It should be noted, however, that the DVD navigationmodule 1605 and content conversion module 1610 may be implemented inhardware, or any combination of hardware and software while stillcomplying with the underlying principles of the invention.

As indicated in FIG. 16 a, in addition to processing converted DVDcontent 1612, the transport stream decoder 1620 may also processtransport streams from one or more alternate sources 1625. The alternatetransport stream sources 1625 may include, for example, cable andsatellite channels received by the multimedia receiver system on whichthe transport stream decoder 1620 is configured (e.g., such as the MediaServer 110 or standard video receiver systems described above).

Referring now to FIG. 16 b, one embodiment of the content conversionmodule 1610 is comprised generally of subpicture processing logic 1680for implementing the subpicture processing techniques described below,and stream processing logic 1681 for implementing the stream conversiontechniques described below.

Stream Processing Logic

As illustrated in FIG. 16 c, in one embodiment, the stream processinglogic 1681 is comprised of repacketizer logic 1683 for convertingprogram stream packets 1675 provided by the DVD navigation module 1605into transport stream packets 1676. In addition, in order to provide anaccurate program clock reference required by the transport streamdecoder 1620, in one embodiment, the stream processing logic 1681includes timestamp processing logic 1682 for converting the System ClockReference (“SCR”) timestamp values 1670 stored within the program streamheaders into Program Clock Reference (“PCR”) values 1671.

FIG. 16 d illustrates one particular method implemented by the streamprocessing logic 1681 and DVD navigation module 1605 to perform thestream conversion. At 1650, the DVD navigation module 1605 extracts theprogram stream from the DVD based on user selections and/or on thenavigation metadata stored on the DVD (e.g., by interpreting the .IFOfiles as described above). At 1651 the repacketizer logic 1683depacketizes the program stream by extracting the underlying audio/videocontent and discarding the program stream headers. In one embodiment,the result of the depacketization process is the Packetized ElementaryStream (“PES”) described above. In addition, the timestamp processinglogic 1682 extracts the SCR timestamp values from the headers of theprogram stream packets.

At 1652, the repacketizer 1683 generates 188-byte transport streampackets using the audio and video content from the PES. In oneembodiment, each video transport packet is assigned a first specifiedpacket ID (“PID”) value and each audio transport packet is assigned asecond specified PID value. The transport stream decoder 1620 uses thesepre-assigned PID values to identify which PID packets contain video andwhich contain audio. In one embodiment, the audio and video PID valuesare maintained, regardless of which particular program stream is beingread from the DVD.

Interleaved video units consist of two or more video and audio streamsof potentially independent timebases broken into chunks and interleavedon the disc as a contiguous stream. In one embodiment, the interleavedpieces are identified through use of DVD metadata (.IFO files and/ornavigation pack data) and packets from each logical stream are assignedto a PID value separate from the other stream. In another embodiment,only the video and audio from the currently playing video isrepacketized. Switching logical streams results in the new stream beingassigned to the PID values in use.

Similarly, the multiple audio and subpicture tracks allowed by a DVD caneither be assigned to separate PID values or be filtered in such a wayas to repacketize only the active tracks into the active PID values.

In addition, the repacketizer logic 1683 periodically inserts PCRtimestamps into the PID packet headers based on the SCR values read fromthe program stream packets. Both the SCR and PCR timestamp values are 33bits in length and both have a resolution of 27 MHz. Accordingly, forconsecutive SCR values representing sequential portions of theaudio/video stream, the conversion process is relativelystraightforward: each consecutive SCR timestamp read from the programstream is replaced with a sequential PCR timestamp in the transportstream.

In some cases, however (determined at 1653 in FIG. 16 d), the SCR values1670 read from the program stream packets 1675 will not be consecutive.This may occur, for example, if the DVD navigation module 1605 jumpsfrom one particular program stream on the DVD to a different programstream (e.g., in response to user input). FIGS. 17 and 18 graphicallyillustrate the differences between certain program stream jumps forwhich timestamps remain consecutive, and jumps for which timestampsbecome non-consecutive. As indicated in FIG. 17, certain DVDs may beencoded with two or more program streams that progress in parallel,using a common timeline (commonly referred to as “seamless angles”). Forexample, the two video streams may represent two different perspectivesof the same football game or scene from a movie. Because the streamsprogress using a common timeline, the SCR timestamps remain consecutive,even when the user jumps from stream S1 to S2 as indicated. Accordingly,the conversion from SCR to PCR values remains relativelystraightforward.

By contrast, as illustrated in FIG. 18, certain program streams may notshare a common timeline (referred to as “non-seamless angles”). Forexample, a DVD may be encoded with a primary video stream (e.g., amovie) and one or more secondary streams (e.g., short presentations ofhow certain scenes of the movie were produced). Because the secondarystreams do not necessarily share a common timeline with the primarystream, when the DVD navigation module 1605 jumps from one stream to thenext (e.g., from S1 to S2 in FIG. 18), the SCR values may becomenon-consecutive.

One problem with non-seamless jumps is that the transport stream decoder1620 may not be capable of properly rendering a stream withnon-consecutive timestamps. For example, on a non-seamless jump, thedecoder 1620 may stop decoding until it receives a packet with the nextconsecutive timestamp (which may never arrive). Accordingly, in oneembodiment, at 1650, when the stream processing logic 1681 detects anon-consecutive jump in the SCR/PCR timestamps, it will activate theresynchronization signal 1616, directing the transport stream decoder toresynchronize itself to the new PCR timeline. Alternatively, or inaddition, in one embodiment, the timestamp processing logic 1682 maysimply continue to provide consecutive PCR values to the transportstream decoder notwithstanding the fact that a non-consecutive jump inthe SCR values has occurred. As long as a consecutive PCR value issubstituted for each SCR value (consecutive or not) the transport streamdecoder 1620 will not need to be resynchronized.

Fast Forward and Rewind

The DVD navigation module 1605 and/or content conversion module 1610 mayemploy additional stream processing techniques to execute fast forwardand rewind functions. As illustrated in FIG. 19, to fast forward orrewind through content stored on a DVD, the DVD navigation module 1605reads only certain specified groups of frames 1900–1903 from the DVD,each separated by a particular amount of time. Different levels offast-forward and rewind are possible (e.g., 2×, 4×, . . . etc), eachcorresponding to a different amount of time between selected frames1900–1903 (e.g., the jump between frames for a 4× fast forward would betwice as large as for a 2× fast-forward). The Data Search Information(“DSI”) embedded within the navigation pack of a VOBU indicates how farthe DVD navigation module 1605 should jump to reach a decodablereference frame for a given fast-forward/rewind rate, and how muchaudio/video content the DVD navigation module 1605 should process oneach jump.

When jumping from frame to frame in this manner, the SCR timestamps 1670associated with each frame will jump accordingly. As such, in oneembodiment of the invention, the content conversion module 1610 willassert the resynchronization signal 1616 on each jump, therebyresynchronizing the transport stream decoder 1620 on each jump asdescribed above. Alternatively, if the transport stream decoder 1620supports a non-synchronized playback mode (many do), the contentconversion module 1610 may simply assert a non-synchronization controlsignal (not shown), causing the transport stream decoder 1620 to decodethe selected frames 1900–1903 without PCR timestamp synchronization.Because fast-forward and rewind functions are not typically executedwith audio, synchronization between the audio and video streams is notnecessarily required.

As illustrated in FIG. 20, it is likely that the boundaries of a pictureextracted from a PES stream will not cleanly align to the PES packetboundary. If that succeeding partial picture is fed to the transportstream decoder, the decoder may become confused and output corruptedvideo. Accordingly, in one embodiment, the repacketizer logic 1683 fillsthe remainder of the final PID packet body 2011 (i.e., following theselected frames) with stuffing bytes 2020, and modifies both the PESheader and the final PID packet header 2010 to indicate where thestuffing bytes 2020 begin. The next frame or group of frames (e.g.,1901) will then start at the beginning of the next packet.

Video Stills

Several video still modes are defined in the DVD specification in whicha designated video image is decoded and held on screen for a period oftime, both with and without an accompanying audio track (e.g., such asthe still image displayed behind the opening DVD menu on many DVDmovies). While standard DVD players/decoders are capable of properlyrendering these still images along with the background audio content,certain problems may result when using a transport stream decoders.

One particular problem and embodiments of a solution will be describedwith respect to FIGS. 21 a and 21 b. FIG. 21 a shows the order in whicha typical sequence of I, B and P frames are read from the DVD andreceived by the transport stream decoder 1620. As illustrated, thetransmission order (e.g., IPBB . . . ) is different from the displayorder (e.g., IBBP . . . ), due to the manner in which the vector-based Band P frames rely earlier and/or later I or P frames. For example, FIG.21 b illustrates a back-predictive buffer 800, a current frame buffer810 and a forward predictive buffer 820 employed during the decodingprocess within the transport stream decoder. The current frame buffer810 contains the frame currently being rendered on the video display. Inthe illustrated example, the transport stream decoder 1620 hassuccessfully received the first I frame and the first P frame. Todisplay the first B frame (which will be the next frame displayed insuccession following the I frame) the initial I frame and the initial Pframe are stored in the back predictive buffer 800 and the forwardpredictive buffer 820, respectively. Accordingly, if the command togenerate the still video image occurs when the buffers are in the stateshown in FIG. 21 b (i.e., at the point indicated in FIG. 21 a), then thecurrent frame buffer will never be filled and no still video image willbe displayed. The transport stream decoder may simply continue to waitfor the B frame to arrive without displaying a video image.

To solve this problem, one embodiment of the invention “forces” thestill image into the current frame buffer 810 and onto the screen bytransmitting a series of “null P frames” to the transport stream decoderafter a command to display a still video image. Null P frames are Pframes in which all motion vectors are equal to zero. As such, the imagegenerated upon receipt of a P frame will be the same as thepreviously-generated image (e.g., the original I frame shown in FIG. 21b).

In one embodiment, the content conversion module 1610 continues totransmit null P frames in succession to the transport stream decoder1620 as the still video image is displayed. Alternatively, after forcingthe still video image into the current frame buffer 810, the contentconversion module 1610 may transmit stuffing bytes within the videostream packets. The determination as to whether to use stuffing bytesmay be based on the particular type of transport stream decoder 1620employed in the system.

Another type of DVD still occurs when one or more consecutive VOBUscontain solely a navigation pack. Since these navigation packs arefiltered out of the stream before being fed to the transport decoder,the decoder receives no input and does not pause. In one embodiment, thecontent conversion module 1610 scans the multimedia stream for thissituation and responsively generates a null P frame that is timestampedwith the PCR (converted SCR) of the navigation pack that was filteredout of the stream. As some implementations of a transport decoder do notrecognize the resulting PCR delta as a still image, another embodimentmay force the decoder into a pause mode, use an external timer to markthe calculated duration of the still and unpause the playback enginewhen the duration has passed. The duration is calculated by subtractingthe SCR of the last submitted media packet (audio or video) from the SCRof the first navigation pack following the media-less (no audio orvideo) VOBU section.

Once the still image is successfully displayed, another problem whichmay result is that the background audio content extracted from the DVDmay begin to accumulate within the transport decoder input buffer. Thisis due to the fact that the video content provided to the transportstream decoder 1620 (i.e., contained via the null P frames) may onlytake up a small fraction of the transport stream relative to a standardvideo signal once the still video image has been generated (null Pframes may be on the order of 200 bytes in length). Because the spaceset aside for the transport decoder input buffer is of a fixed size, theaudio may begin to take up a disproportionate share of the input buffer,resulting in large latencies (i.e., several seconds) in data submission,playback and stream control.

In order to prevent this from happening, in one embodiment, the contentconversion module 1610 spaces the audio segments out by a predeterminedamount using stuffing packets. The stuffing packets are simply discardedby the transport stream decoder 1620. In one embodiment, the number ofstuffing packets generated by the content conversion module 1610 isbased on the bitrate of the audio content. In one embodiment, thecontent conversion module 1610 parses the headers of the AC-3 packetscontained within the audio stream to determine the approximate audiobitrate (as defined by the DVD specification, AC-3 packet headersinclude an indirect indication of bitrate through thenumber_of_frame_headers field).

AC-3 Audio Conversion

The AC-3 format used for DVDs is slightly different than the AC-3 formatprocessed by transport stream decoders. In particular, as illustrated inFIG. 22, an additional AC-3 header 2230 is employed within the DVDprogram stream 2200 which a standard DVD player is capable ofprocessing, but which a transport decoder is not. The PES header 2210contains a field (PES_header_data_length) which points to the beginningof the packet payload. In a DVD stream, the packet payload begins withthe additional AC-3 header described 2230. Accordingly, in oneembodiment, in order to bypass the AC-3 header information, the contentconversion module 1610 modifies the PES_header_data_length field toindicate that the packet payload begins at the start of the actual AC-3audio content rather than the AC-3 header.

Subpicture Processing

Referring again to FIG. 15, each VOBU 1530 may include subpicture datapackets 1543 multiplexed with audio and video program stream packets1541 and 1542, respectively. Subpictures are graphical imagessuperimposed over the decoded video images. Subpictures are used toprovide subtitles and to construct user interface features such asinteractive navigation menus. Each subpicture data packet 1543 includesa header, a pixel data block, and a command data block. The headeridentifies the packet as a subpicture data packet. The pixel data blockcontains run-length encoded (“RLE”) bitmap image data. The command datablock identifies the location and size of the subpicture. Additionaldata contained within the Presentation Control Information (PCI) blockof a navigation pack specifies the location and size of highlight areaswithin the subpicture. Highlight areas are one or more smaller areaswithin the subpicture that may have unique color and contrast (i.e.,alpha blending) values applied. A color lookup table is separatelyextracted from the .IFO data files and provides a palette of color andalpha values that are legal to use during the valid period of thatpalette.

As illustrated in FIG. 23, the subpicture processing module 1680generates subpicture images by interpreting the RLE bitmap data 2300 inlight of color palette data 2302, and highlight data 2301. The colorpalette data 2302 is provided from the .IFO file(s) and the highlightdata 2301 is transmitted as Presentation Control Information (“PCI”).The highlight data 2301 identifies certain regions of the subpicturewhich should be highlighted (e.g., such as a selected menu item). Adifferent portion of the color palette data 2302 is specified forregions of the subpicture which are highlighted.

An alpha value (α), which defines the translucence level, is associatedwith each pixel in the subpicture. The α values range from completelytranslucent (α=0) to complete opaque (α=100%). The RLE bitmap 2300 maydescribe all or a portion of a full video screen of alpha-blendedgraphics, notwithstanding the fact that certain regions of the screen(e.g., regions 2310 in FIG. 23) are completely translucent. Thus, togenerate subpicture images, standard subpicture processors redraw theentire video screen without consideration of the level of translucenceor whether some or all of the non-translucent pixel data has changedfrom one picture to the next. Continually redrawing an entire screen ofgraphical content in this manner wastes a significant amount of memoryand CPU processing power.

Accordingly, in one embodiment of the invention, the subpictureprocessing module 1680 performs an analysis of the various graphicaldata 2301–2302 provided to it, and only draws regions of the sub-picturewhich are not translucent (i.e., regions such as region 2310 for whichfor which α>0) or are newly translucent (i.e., has been “erased” fromone picture to the next). In addition, once an initial subpicture isdrawn, one embodiment of the subpicture processing module 1680 onlyredraws areas which are subsequently modified (e.g., as a result ofmotion of the highlight region 2320).

One embodiment of a method implemented by the subpicture processingmodule 1680 is outlined in FIG. 24. At 2400, the subpicture processingmodule 1680 examines the incoming RLE subpicture data in conjunctionwith the subpicture palette and/or highlight data to identify regions ofthe screen for which α=0. In one embodiment, these regions can bequickly identified based on the existence of an α=0 indication inconjunction with a “repeat until end of line” indication (i.e., the “endof line” indicator (fourteen sequential zeros) followed by a paletteentry code corresponding to fully transparent alpha; subpictures aredefined on a line-by-line basis). This scanning process may beimplemented in a highly efficient manner on processor architectures thatsupport hardware-assisted zero run length counting (e.g., as describedby the PowerPC cntlzw mnemonic). Another implementation may performadditional processing of subpicture RLE data to more fully constrain thetranslucent and non-translucent regions.

Once the transparent regions have been identified, the subpictureprocessing module 1680 filters them out of the subpicture generationprocess and generates subpicture images for the non-transparent regionsonly, at 2405, thereby conserving memory and CPU cycles.

At 2410, the subpicture processing module 1680 determines whether thegraphical image has changed relative to the current subpicture bitmap.For example, if a user selects a particular menu item, a new menustructure may be displayed further down the menu hierarchy. Similarly,it may be time for the subpicture processing module 1680 to display anew subtitle. In any case, when it is time to generate a new bitmapdata, at 2412 the subpicture processing module 1680 identifies andaccumulates only those regions of the image affected by the new data.

In addition, at 2415, the subpicture processing module 1680 monitorsuser input to determine whether the highlight region 2320 has beenmoved. As illustrated in FIG. 25, motion of the highlight region meansthat the palette data 2302 applied to both the newly-highlighted region2500 and the previously-highlighted region 2501 will change (i.e., thepreviously-highlighted region 2501 will use the same portion of thepalette data 2302 as the rest of the subpicture and thenewly-highlighted region 2500 will use the portion of the palette data2302 designated for highlighting). Rather than redrawing the entirenon-translucent region, however, at 2417, the subpicture processingmodule 1680 accumulates only those regions affected by the highlightmovement (i.e., on a line-by-line basis). That is, only that portion ofthe subpicture which is either now within the highlight rectangle (e.g.,region 2500) or has just come out from a highlight rectangle (e.g.,region 2501). Finally, at 2420, the subpicture processing module 1680only redraws the changed regions accumulated at 2412 and 2417, therebyconserving memory and CPU cycles.

It should be noted that throughout the foregoing description, well-knownstructures and devices have been shown in block diagram form to avoidobscuring the underlying principles of the invention. For example, themechanical and optical hardware required to read the content from a DVDis well known in the art and, accordingly, is not illustrated ordescribed above (e.g., with respect to the DVD navigation module 1605).Similarly, the hardware required to superimpose a graphical subpictureimage over a video image it is well understood by those of ordinaryskill in the art and, therefore, is not illustrated or described.Various other unnecessary details have been omitted to avoid obscuringthe principles of the invention.

Embodiments of the present invention include various steps, which havebeen described above. The steps may be embodied in machine-executableinstructions which may be used to cause a general-purpose orspecial-purpose processor to perform the steps. Alternatively, thesesteps may be performed by specific hardware components that containhardwired logic for performing the steps, or by any combination ofprogrammed computer components and custom hardware components.

Elements of the present invention may also be provided as a computerprogram product which may include a machine-readable medium havingstored thereon instructions which may be used to program a computer (orother electronic device) to perform a process. The machine-readablemedium may include, but is not limited to, floppy diskettes, opticaldisks, CD-ROMs, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs,magnet or optical cards, propagation media or other type ofmedia/machine-readable medium suitable for storing electronicinstructions. For example, the present invention may be downloaded as acomputer program product, wherein the program may be transferred from aremote computer (e.g., a server) to a requesting computer (e.g., aclient) by way of data signals embodied in a carrier wave or otherpropagation medium via a communication link (e.g., a modem or networkconnection).

Throughout the foregoing description, for the purposes of explanation,numerous specific details were set forth in order to provide a thoroughunderstanding of the present system and method. It will be apparent,however, to one skilled in the art that the system and method may bepracticed without some of these specific details. For example, while thestream conversion techniques described above focus on converting DVDprogram streams to transport streams, the underlying principles may beimplemented using a variety of alternate program stream sources.Similarly, while described above in the context of a “Media Server,” theunderlying principles of the invention may be implemented on any systemconfigured with a transport processor and a DVD reader. Accordingly, thescope and spirit of the invention should be judged in terms of theclaims which follow.

1. A method for rendering subpicture images comprising: reading asubpicture image from a digital versatile disk (“DVD”); identifyingregions within said subpicture image for which a translucence level, α,is below a specified threshold value, wherein identifying furthercomprises locating, within a subpicture image buffer, an indication thatα is below said specified threshold value, and an indication that said αvalue repeats until an end of a current line of said subpicture image;and rendering on a video screen only those regions of said subpictureimage for which α is above said specified threshold value.
 2. The methodas in claim 1 wherein said specified threshold value is zero.
 3. Themethod as in claim 1 wherein said indication comprises fourteensequential zeros followed by a palette entry code corresponding to α=0.4. The method as in claim 1 implemented by a processor which supportshardware-assisted zero run length counting.
 5. The method as in claim 1further comprising: detecting that one or more regions of saidsubpicture image have changed; and regenerating on said video screenonly said regions of said subpicture image which have changed, whilemaintaining all remaining regions.
 6. The method as in claim 5 whereinsaid changed regions are comprised of new bitmap data.
 7. The method asin claim 5 wherein said changed regions are comprised of new palettedata.
 8. The method as in claim 1 further comprising: detecting that ahighlight associated with a first region has been moved to a secondregion on said subpicture image; and regenerating on said video screenonly said first region without said highlight and said second regionwith said highlight, while maintaining all remaining regions.
 9. Themethod as in claim 8 wherein said highlight is moved responsive to auser selection input via a remote control device.
 10. A tangiblecomputer readable medium, other than a carrier wave or other propagationmedium, including computer program code which, when executed by amachine, causes said machine to perform the operations of: reading asubpicture image from a digital versatile disk (“DVD”); identifyingregions within said subpicture image for which a translucence level, α,is below a specified threshold value, wherein identifying furthercomprises locating, within a subpicture image buffer, an indication thatα is below said specified threshold value, and an indication that said αvalue repeats until an end of a current line of said subpicture image;and rendering on a video screen only those regions of said subpictureimage for which α is above said specified threshold value.
 11. Thecomputer readable medium as in claim 10 wherein said specified thresholdvalue is zero.
 12. The computer readable medium as in claim 10 whereinsaid indication comprises fourteen sequential zeros followed by apalette entry code corresponding to α=0.
 13. The computer readablemedium as in claim 10 including additional program code to cause saidmachine to perform the operations of: detecting that one or more regionsof said subpicture image have changed; and regenerating on said videoscreen only said regions of said subpicture image which have changed,while maintaining all remaining regions.
 14. The computer readablemedium as in claim 13 wherein said changed regions are comprised of newbitmap data.
 15. The computer readable medium as in claim 13 whereinsaid changed regions are comprised of new palette data.
 16. The computerreadable medium as in claim 10 including additional program code tocause said machine to perform the operations of: detecting that ahighlight associated with a first region has been moved to a secondregion on said subpicture image; and regenerating on said video screenonly said first region without said highlight and said second regionwith said highlight, while maintaining all remaining regions.
 17. Thecomputer readable medium as in claim 16 wherein said highlight is movedresponsive to a user selection input via a remote control device.
 18. Asystem comprising: DVD reader means for reading a subpicture image froma digital versatile disk (“DVD”); image analysis means for identifyingregions within said subpicture image for which a translucence level, α,is below a specified threshold value, wherein said image analysis meansidentifies said regions by locating, within a subpicture image buffer,an indication that α is below said specified threshold value, and anindication that said α value repeats until an end of a current line ofsaid subpicture image; and image rendering means for rendering on avideo screen only those regions of said subpicture image for which α isabove said specified threshold value.
 19. The system as in claim 18wherein said specified threshold value is zero.
 20. The system as inclaim 18 wherein said indication comprises fourteen sequential zerosfollowed by a palette entry code corresponding to α=0.
 21. The system asin claim 18 wherein said image analysis means detects that one or moreregions of said subpicture image have changed and responsivelyregenerates on said video screen only said regions of said subpictureimage which have changed, while maintaining all remaining regions. 22.The system as in claim 21 wherein said changed regions are comprised ofnew bitmap data.
 23. The system as in claim 21 wherein said changedregions are comprised of new palette data.
 24. The system as in claim 18wherein said image analysis means detects that a highlight associatedwith a first region has been moved to a second region on said subpictureimage; and regenerates on said video screen only said first regionwithout said highlight and said second region with said highlight, whilemaintaining all remaining regions.
 25. The system as in claim 24 whereinsaid highlight is moved responsive to a user selection input via aremote control device.